Study of Classification Algorithms using Moment Analysis

In this short paper we briefly discuss a moment based method that was recently introduced to study the behavior classification algorithms and model validation techniques for finite sample sizes. The method involves accurate and efficient computation of the moments of the generalization error which are over the space of all possible datasets of size N drawn from an underlying distribution. A classification algorithm trained on each of these datasets induces a space of classifiers (i.e. an empirical hypothesis space) and the moments can be equivalently computed over this space. In our previous work we also drew relationships between the moments of the generalization error and moments of hold out error, cross-validation error, leave one out error and hence these model validation techniques can also be studied accurately by our method. The primary goal of this paper is to familiarize machine learning researchers with this newly proposed methodology, so as to discuss its implications regarding important problems such as classification model selection.